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DETAILED ACTION 
Response to Amendment 

1. Upon entry of Request for Continued Examination filed on May 07, 2004, the 
amendment filed on April 02, 2004 has been entered. 

2. Claims 1-21 are pending. 

Response to Arguments 

3. Applicant's arguments, see Remarks, pages 8-10, filed on April 02, 2004, with respect to 
the rejection(s)of claim(s) 1, 20 and 21 under 35 U.S. C. 103 (a) have been fully considered and 
are persuasive. Therefore, the rejection has been withdrawn. However, upon further 
consideration, a new ground(s) of rejection is made in view of Colmenarez et al ("Embedded 
face and facial expression recognition", Image Processing, 1999. ICIP 99. Proceedings. 1999 
International Conference on, Volume: 1, 1999 Pages: 633 - 637 vol.1). 

Claim Rejections - 35 USC § 102 
The following is a quotation of the appropriate paragraphs of 35 U.S. C. 102 that form the 

basis for the rejections under this section made in this Office action: 
A person shall be entitled to a patent unless - 

(b) the invention was patented or described in a printed publication in this or a foreign country or in public use or on 
sale in this country, more than one year prior to the date of application for patent in the United States. 

4. Claims 1-3, 5-8 and 18-21 are rejected under 35 U.S.C. 102(b) as being anticipated by 
Colmenarez et al ("Embedded face and facial expression recognition", Image Processing, 1999. 
ICIP 99. Proceedings. 1999 International Conference on, Volume: 1, 1999 Pages: 633 - 637 
vol.1). 
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a. Referring to Claim 1, Colmenarez discloses processing a sequence of images 
("video sequence", page 633, right column, lines 17-18) to generate a statistical model for 
each person to be tagged (page 634, left column, equation 2), the statistical model 
incorporating at least one appearance feature ("feature appearance", page 633, right 
column, paragraph 5) and at least one geometric feature ("feature positions", page 633, 
right column, paragraph 5") of the tagged person; applying the model to at least one 
subsequent image in order to perform at least one of a detection operation, a location 
operation and a tracking operation for the tagged person (the statistical model is applied 
in facial expression recognition, which is a detection operation. Therefore, at least one of 
the alternative limitations is satisfied), wherein an optimum trajectory of movement (the 
optimum trajectory of movement in Colmenarez is the facial feature movement in the 
facial expression demonstrated by the movement of the mouth, eyes, and eyebrows in the 
facial expressions of figure 3, which the statistical model of the facial feature is 
maximized, see page 634, left column, equation 3) of a tagged person is maximized 
through linear transformation (transformation matrix T, page 635, left column, line 22); 
and controlling an action of the image processing system based on a result of the at least 
one operation ("facial expression" is the claimed "an action", which is determined based 
on the facial recognition algorithm, page 635, right column, lines 12-24). 

b. Referring to Claim 2, Colmenarez discloses wherein the sequence of image 
comprises a video segment (page 635, right column, lines 7-11). 

c. Referring to Claim 3, Colmenarez discloses wherein the processing step further 
includes processing the sequence of images to generate a plurality of statistical models, 



Application/Control Number: 09/703,423 Page 4 

Art Unit: 2621 

each of the models corresponding to a particular tagged person (page 635, right column, 
last full paragraph). 

d. Referring to Claim 5, Colmenarez discloses wherein the geometric feature 
comprises at least one of a region shape and a region position for given one of a plurality 
of regions associated with the statistical model (page 634, right column, paragraph 2, 
position x ki ). 

e. Referring to Claim 6, Colmenarez discloses wherein the statistical model is 
generated at least in part by segmenting a given image into a number N of different 
regions of similar appearance (Note the segmenting in figures 2 and 3. Note also that 
there are 2 (i.e. N) different regions for the eyes (similar appearance), 2 difference 
regions for the left eyebrow, 2 for the right eyebrow, etc). 

f Referring to Claim 7, Colmenarez discloses wherein the statistical model 
generated for a given person comprises a likelihood probability function, which indicates 
the likelihood that the person is present in a given image (page 634, see equation 2). 

g. Referring to Claim 8, Colmenarez discloses wherein the likelihood probability 

function P(I | Q) for person His computed as P(I \ Q) = P(/? r | Q)P(r \ Q) , where 

r=\,2,...N 

R r is a function of the at least one appearance feature and the at least one geometric 
feature, and r is an index identifying one of N regions of similar appearance within the 
image I (page 634, see equation 2, which is identical to the claimed function except for 
the specific symbol used). 

h. Referring to Claim 18, Colmenarez discloses wherein the controlling step 
comprises generating an output of the image processing system based on the result of the 
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at least one operation (the facial expression of the person is determined, page 635, right 
column, paragraph 2). 

i. Referring to Claim 19, Colmenarez discloses wherein the controlling step 
comprises altering an operating parameter of the image processing system based on the 
result of the at least one operation (the hidden variable e , the operating parameter herein, 
is replaced by a hidden state in a HMM, which the variable is altered between facial 
expressions in a sequence of video frames, see page 636, right column, last paragraph), 
j. With regard to Claim 20, all the limitations are addressed in Claim 1 . 
k. With regard to Claim 21, the only difference between Claim 1 and Claim 21 is 
Claim 21 calls for additional limitation of a storage medium for storing one or more 
programs, which Colmenarez teaches using a computer system to implement his 
invention, which inherently requires a storage medium for storing one or more programs. 

Claim Rejections - 35 USC § 103 
The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set forth in 
section 102 of this title, if the differences between the subject matter sought to be patented and the prior art are 
such that the subject matter as a whole would have been obvious at the time the invention was made to a person 
having ordinary skill in the art to which said subject matter pertains. Patentability shall not be negatived by the 
manner in which the invention was made. 

5. Claim 4 is rejected under 35 U.S.C. 103(a) as being unpatentable over Colmenarez in 

view of Darrell et al (U.S. Patent No. 6,188,777 Bl). The arguments in Paragraph 4. a above as to 

the applicability of Colmenarez are incorporated herein. Colmenarez already discloses the 

appearance feature is incorporated into the statistical model. However, Colmenarez does not 

explicitly teach the appearance feature is at least one of a color feature and a texture feature. 
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However, Darrell at column 4, lines 28-29, teaches using color detection and segmentation 
module to detect regions of fresh tone in a target region, the target region is a face, column 2, 
line 51. At the time the invention was made, it would have been obvious to a person of ordinary 
skill in the art to use the color feature as appearance feature. A person of ordinary skill in the art 
would have been motivated to do this because both Colmenarez and Darrell teaches using 
appearance features for facial recognition/tracking, and it would have been an obvious matter of 
design choice to use the color feature as appearance feature in Colmenarez since applicant has 
not disclosed that using a color feature would post any significant advantages and it appears that 
the facial recognition algorithm would perform equally well with any appearance feature. 
6. Claims 9-14 are rejected under 35 U.S.C. 103(a) as being unpatentable over Colmenarez 
in view of Chang et al (U.S. Patent No. 5,999,651). The arguments in Paragraph 4. a above as to 
the applicability of Colmenarez are incorporated herein. 

a. Referring to Claim 9, Colmenarez discloses wherein the statistical model 
generated for a given person comprises a likelihood probability function 
P(r k \s,p), where a; is feature region as shown in figure 2, and sis the claimed 

discrete variable used t capture local motion, and pis the person in the image. 
Although Colmenarez does disclose a linear transform matrix T at page 635, left 
column for local motion, Colmenarez does not explicitly teach a linear 
transformation that is used to capture global motion of a person in the image. 
Chang teaches predicting global motion by using a linear transformation (at 
column 4, line 28) before estimating local motion. At the time the invention was 
made, a person of ordinary skill in the art would have been motivated to 
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incorporate a linear transformation for global motion in a facial recognition model 
like Colmenarez's because Chang at column 4, lines 17-19 teaches global motion 
prediction provides a good initial estimate for the subsequent motion estimation 
algorithm, and it would be reasonable to a person of ordinary skill in the art to 
modify a system like Colmenarez's to obtain the global movement of a person's 
face in a sequence of video images before focusing on particular facial features to 
determine the facial expression. 

b. Referring to Claim 10, the combination of Colmenarez and Chang discloses 
wherein a location of the person of determined using the linear transformation 
(see Chang, column 4, line 29). 

c. Referring to Claim 11, the combination of Colmenarez and Chang discloses 
wherein a pose of the person is determined using the discrete variable £(see 
explanation in Claim 9 for discrete variable in Colmenarez, and the discrete 
variable is used to capture the local motion of the facial features, which 
determines the facial expression of a person, such as sadness, happiness . . . etc, a 
facial expression is a pose). 

d. Referring to Claim 12, the combination of Colmenarez and Chang teaches 
wherein the linear transformation is used to obtain a sub-window of the image I 
that is invariant to rotation and scale (see figure 2 in Colmenarez and the 
explanation in paragraph e below). 

e. Referring to Claim 13, the combination of Colmenarez and Chang teaches 
wherein the linear transformation T is implemented using bilinear interpolation 
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technique with a reference point in the image, a rotation angle and a scaling factor 
(Chang in figures l-2b teaches using two video frame to detect global motion, 
which uses a bilinear interpolation technique, and using affine motion correction 
to determine the rotation and scaling factor of the frame for motion error 
measure). 

f Referring to Claim 14, the combination of Colmenarez and Chang teaches 
wherein the local motion is modeled using a discrete set of states {£ 15 £ 2 °f 
the variable £to capture M different poses of the person (see figure 3 in 
Colmenarez, Colmenarez discloses tracking six different facial expressions in a 
sequence of video images, each facial expression in an video image is a pose). 
Allowable Subject Matter 

7. Claims 15-17 are allowed. 

The following is a statement of reasons for the indication of allowable subject matter: 
Independent claim 15 defines a statistical model generated for a given person Qand 

image I comprises a likelihood probability function P(I \ 7\£,Q) = ^P(pix | 7\£,n) , where r 

pixel 

is an index to regions of similar appearance and TV is a total number of such regions, r=J,2 t ...N, 
and P(pix \r, T,%,Q) = max [P(pix | r, 7 1 , Q)P(r | Q)] ,where P(pix \ r, T, Q) is the 
probability of observing pixel pix assuming that it belongs to an r-th region of the model on a 
pose £ , and P(r | Q) is the prior probability of the region at that pose. This feature in 
combination with other features in Claim 15, which is the broadest allowable claim, is not taught 
or suggested by the art of record. 
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Claims 16-17 are dependent upon Claim 15. 

Conclusion 

8. The prior art made of record and not relied upon is considered pertinent to applicant's 
disclosure. 

a. "Face detection with information-based maximum discrimination", Colmenarez, 
A J.; Huang, T. S. /Computer Vision and Pattern Recognition, 1997. Proceedings., 

1997 IEEE Computer Society Conference on , 17-19 June 1997, Pages: 782 - 
787. see section 4, face detection system and figure 2. 

b. "A probabilistic framework for embedded face and facial expression recognition", 
Colmenarez, A.; Frey, B.; Huang, 7.5. /Computer Vision and Pattern Recognition, 
1999. IEEE Computer Society Conference on. , Volume: 1,23-25 June 1999. 
Pages: 597 Vol 1. see sections 2 and 3. 

c. "Mixtures of local linear subspaces for face recognition" Frey, B.J.; Colmenarez, 
A.; Huang, T.S.; Computer Vision and Pattern Recognition, 1998. Proceedings. 

1998 IEEE Computer Society Conference on , 23-25 June 1998. Pages:32 - 37. 
see sections 3 and 4. 

d. "Detection and tracking of faces and facial features", Colmenarez, A.; Frey, B.; 
Huang, r.£;Image Processing, 1999. ICIP 99. Proceedings. 1999 International 
Conference on , Volume: 1 , 1999. Pages:657 - 661 vol.1, see section 3 for global 
motion tracking. 
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9. Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to Tom Y Lu whose telephone number is (703) 306-4057. The 
examiner can normally be reached on 8:30AM-5PM. 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Leo H Boudreau can be reached on (703) 305-4706. The fax phone number for the 
organization where this application or proceeding is assigned is 703-872-9306. 

Information regarding the status of an application may be obtained from the Patent 
Application Information Retrieval (PAIR) system. Status information for published applications 
may be obtained from either Private PAIR or Public PAIR. Status information for unpublished 
applications is available through Private PAIR only. For more information about the PAIR 
system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR 
system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). 



Tom Y. Lu 




